Skill Improvement ($50-150 Bounty)

Skill Modified

Skill name: siem-rules
Skill path: skills/secops/siem-rules/

What Was Wrong

The skill asked authors to explain how a SIEM rule produces a true positive, but it did not require replayable validation evidence. A rule could be marked ready after a short manual query run even if it had no true-positive fixture, no benign negative case, no threshold boundary checks, and no regression evidence after tuning changes.

This leaves a practical detection engineering gap: threshold, suppression, lookup, parser, or query-optimization changes can silently remove true positives or reintroduce known false positives.

Related review issue: #2218

What This PR Fixes

• Bumps siem-rules to 1.0.1.
• Adds a new Fixture-Backed Validation step before lifecycle promotion.
• Requires true-positive, benign/false-positive, boundary, schema-variant, and regression evidence.
• Extends the output template with validation status and fixture/replay fields.
• Updates the validation pitfall to require replayable true and negative test cases.
• Adds skill-local JSON fixtures for a privileged off-hours true positive and an approved maintenance/service-principal negative case.

Evidence

Before (skill could treat this as validated):

SigninLogs
| where TimeGenerated > ago(24h)
| where ResultType == 0
| where UserPrincipalName has_any ("admin", "break-glass")
| extend HourOfDay = hourofday(TimeGenerated)
| where HourOfDay < 7 or HourOfDay >= 19

Manual validation over a clean 24-hour window can produce no alerts without proving the rule fires on representative malicious activity or avoids known benign off-hours maintenance.

After (now explicitly handled):

### Validation
- **True-positive fixtures:** [sample path or replay job, expected alert count, key matched entities]
- **Benign/negative fixtures:** [sample path or replay job, expected zero-alert cases]
- **Boundary tests:** [below/at/above threshold or window expectations]
- **Schema variants tested:** [field aliases, parser/connector version, supported backend]
- **Regression result:** [before/after expected alert counts for rule changes]

Test Cases Added/Updated

• Added vulnerable test cases (tests/vulnerable/)
• Added benign test cases (tests/benign/)
• Existing tests still pass

Validation

• git diff --check
• Frontmatter required-field check matching .github/workflows/lint-skills.yml
• JSON parse check for both fixture files
• Markdown fence-balance check for skills/secops/siem-rules/SKILL.md
• Marker check for fixture-backed validation, validation status, true-positive fixtures, benign/negative fixtures, regression result, and expected alert count fields
• Prompt-injection pattern scan over changed skill content

Bounty Tier

• Minor ($50) - Doc update, small logic tweak, typo fix
• Moderate ($100) - New edge case coverage, FP reduction with evidence
• Substantial ($150) - Rewritten detection logic, major coverage expansion

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: GitHub Sponsors or PayPal after maintainer acceptance